To make SCM-Manager more accessible and to make it easier using scripts against the server, we created a command line interface. This command line interface can be used to perform the default actions like create, modify and delete repositories. It is also very flexible and can be extended by plugins.
The CLI already supports internationalization, help texts, input validation, loose and table-like templates and nested subcommands. Check the cli guidelines to learn how add new cli commands.
Co-authored-by: Sebastian Sdorra <sebastian.sdorra@cloudogu.com>
Enable gradle dependency locking for all subprojects. This change allows us to use dependency ranges, without loosing reproducible builds.
It also makes it easier for third party services such as snyk to detect dependencies and their versions.
Users, groups, repositories and repository roles have been sorted in the rest layer by default if no other sort option was given. In the layers "below" (aka the manager classes or the dao), the collections have been unsorted. This led to the effect, that the autocomplete resource, which did not sort all values beforehand, returned unsorted results. As a sideeffect, direct matches for an input could occur at a random position or not at all (as reported in #1695), when there were enough other matches.
With this pull request the databases for users, groups, repositories and repository roles will use instances of TreeMap instead of LinkedHashMap internally, so that these values are sorted implicitly (by id respectively name for users, groups and repository roles and namespace/name for repositories).
Due to this change the default sort applied in the rest layer could be removed.
Update Gradle to 7.3. in order to profit from the performance improvement and to prepare for dependency lockfiles. Due to the upgrade to 7.x we had to use a license plugin which supports Gradle 7.
On windows unit tests are failing because junit checks if all @tempdir directries are empty and can be deleted after test run.
But due to opened file handles (not closed resource streams) Windows keeps files, which are "in use".
Linux is less strict in this area.
Additionally I want highlight that XMLStreamReaderImpl/XMLStreamWriterImpl from apache.xerces library (in OpenJDK11 at least) which are picked at runtime as xml parser implementation - they don't close associated resources.
BTW, I thing that relying on some runtime (sometimes - unpredictable) dependencies - is bad practice, but this it up to separate topic.
Additional fix: in IOUtil is file is locked (due to permissions or opened handle) - it will undlessly try-and-retry to delete it until end of the world, on windows.
Before this change the search uses a single index which distinguishes types (repositories, users, etc.) with a field (_type).
But it has turned out that this could lead to problems, in particular if different types have the same field and uses different analyzers for those fields. The following links show even more problems of a combined index:
https://www.elastic.co/blog/index-vs-typehttps://www.elastic.co/guide/en/elasticsearch/reference/6.0/removal-of-types.html
With this change every type becomes its own index and the SearchEngine gets an api to modify multiple indices at once to remove all documents from all indices, which are related to a specific repository, for example.
The search uses another new api to coordinate the indexing, the central work queue.
The central work queue is able to coordinate long-running or resource intensive tasks. It is able to run tasks in parallel, but can also run tasks which targets the same resources in sequence. The queue is also persistent and can restore queued tasks after restart.
Co-authored-by: Konstantin Schaper <konstantin.schaper@cloudogu.com>
Add option to encrypt repository exports with a password and add possibility to decrypt them on repository import. Also make the repository export asynchronous. This implies that the repository export will be created on the server and can be downloaded multiple times. The repository export will be deleted automatically 10 days after creation.
The default (XML) store of SCM-Manager does not distinguish between config and config entry stores in regards to
storage locations. Nonetheless, we want to make a difference in export files, so that other store providers can handle
these stores differently. To do so, this change adds an attribute to the top level xml element of config entry stores
to mark them. In exports, these store files can now be exported in a different folder. To mark existing stores, this
introduces an update step.
* Add store exporter to collect the repository metadata
* Add EnvironmentInformationXmlGenerator
* Collect export data and put into compressed tar archive output stream
* Create full repository export endpoint.
* Add full repository export to ui
* Ignore irrelevant files from config store directory
* write metadata stores to file since a baos could teardown the server memory
* Migrate store name for git lfs files (#1504)
Changes the directory name for the git LFS blob store by
removing the repository id from the store name.
This is necessary for im- and exports of lfs blob stores,
because the original name had the repository id as a part
of it and therefore the old store would not be found when
the repository is imported with another id.
Existing blob files will be moved to the new store location
by an update step.
Co-authored-by: Eduard Heimbuch <eduard.heimbuch@cloudogu.com>
* Introduce util for migrations (#1505)
With this util it is more simple to rename
or delete stores.
* Rename files in export
Co-authored-by: René Pfeuffer <rene.pfeuffer@cloudogu.com>
This adds a flag "archived" to repositories. Repositories marked with this can no longer be modified in any way. To do this, we switch to a new version of Shiro Static Permissions (sdorra/shiro-static-permissions#4) and specify a permission guard to check for every permission request, whether the repository in question is archived or not. Further we implement checks in stores and other activies so that no writing request may be executed by mistake.
Co-authored-by: Eduard Heimbuch <eduard.heimbuch@cloudogu.com>
