Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

K-HDTCat #181

Merged
merged 2 commits into from
Dec 1, 2022
Merged

K-HDTCat #181

merged 2 commits into from
Dec 1, 2022

Conversation

ate47
Copy link
Contributor

@ate47 ate47 commented Dec 1, 2022

Hello!

In this PR, I've implemented a version of HDTCat working whatever the input count of HDTs (for MS/4S Dictionaries), it follows the same algorithm as HDTCat_let's_make_HDT_scale, but with multiple HDT input, except that the streams from each HDT are merged to behave like one HDT

I've added it in HDTCatTree with the option loader.cattree.kcat to set the number of HDT to merge at the same time

3 options are here to describe the cat:

  • hdtcat.location: the build directory, by default a temporary location is specified
  • hdtcat.location.future: the future location of the HDT, can't be in the build directory
  • hdtcat.deleteLocation: if we should delete the build directory (if specified)

API

I've added all the options in the HDTOptionsKeys class

A new method was added in HDTManager

public static HDT catHDT(List<String> hdtFileNames, HDTOptions hdtFormat, ProgressListener listener) throws IOException

Using the old method will use the legacy algorithm

CLI

I've added the BATCH version of hdtCat with the hdtCat.bat

I've added the -kcat argument to HDTCat to allow to use the new version of HDTCat with 2 HDTs, but now we can specify the number of HDTs we want

With hdtVerify, I've removed the limit of 1 hdt, I've added 2 arguments:

  • -progress to see the progress of the verify for the current HDT
  • -equals to see that all the HDTs specified are equals

Core

The implementation of HDTCat, with tests

@D063520
Copy link
Contributor

D063520 commented Dec 1, 2022

yea! nice!!!

@D063520 D063520 merged commit d8d4b2a into rdfhdt:master Dec 1, 2022
@ate47 ate47 mentioned this pull request Dec 12, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants