Add security notifications in SCM-Manager to inform running instances about known security issues. These alerts can be core or plugin specific and will be shown to every user in the header.
Co-authored-by: Matthias Thieroff <matthias.thieroff@cloudogu.com>
Co-authored-by: Philipp Ahrendt <philipp.ahrendt@cloudogu.com>
Co-authored-by: Sebastian Sdorra <sebastian.sdorra@cloudogu.com>
Users, groups, repositories and repository roles have been sorted in the rest layer by default if no other sort option was given. In the layers "below" (aka the manager classes or the dao), the collections have been unsorted. This led to the effect, that the autocomplete resource, which did not sort all values beforehand, returned unsorted results. As a sideeffect, direct matches for an input could occur at a random position or not at all (as reported in #1695), when there were enough other matches.
With this pull request the databases for users, groups, repositories and repository roles will use instances of TreeMap instead of LinkedHashMap internally, so that these values are sorted implicitly (by id respectively name for users, groups and repository roles and namespace/name for repositories).
Due to this change the default sort applied in the rest layer could be removed.
Changes the "namespace" input in the repository creation form or the "rename repository" dialog to an
autocomplete input. Of course this is only available for the "custom" namespace strategy.
Fix syntax highlighting on non highlighted fields. Fix ellipsis on new lines in code syntax highlighting. Fix ellipsis on content start or end in non code fields.
Co-authored-by: Sebastian Sdorra <sebastian.sdorra@cloudogu.com>
Updated spotter to version 4 in order to get prism syntax mode for detected coding languages.
Expose syntax modes of coding languages as headers on content endpoint and as fields on diff dto.
Remove leading line break on search result fragments.
Use mark instead of span or strong for highlighted search results.
Add option to use syntax highlighting in TextHitField component.
Co-authored-by: Matthias Thieroff <matthias.thieroff@cloudogu.com>
Allows scm-manager instances to authenticate with the configured plugin center. If the default plugin center is used, a myCloudogu account is used for authentication which in turn enables downloading special myCloudogu plugins directly through the plugin administration page.
Co-authored-by: Konstantin Schaper <konstantin.schaper@cloudogu.com>
Co-authored-by: Matthias Thieroff <93515444+mthieroff@users.noreply.github.com>
Co-authored-by: Philipp Ahrendt <philipp.ahrendt@cloudogu.com>
Fix access to ScmPathInfoStore as Provider to prevent injection errors. Errors might happen, if beans using ScmPathInfoStore (like the BranchLinkProvider) are injected in beans that are singletons or that have be created outside of request scope otherwise. We do so knowing that this might lead to runtime errors (eg. when links shall be build in a thread that has no request scope). We have decided nonetheless that injection errors on startup are worse for plugin developers, that may have no clue how to solve injection problems.
Enrich branch overview with more details like last committer and ahead/behind commits. Since calculating this information is pretty intense, we request it in chunks to prevent very long loading times. Also we cache the results in frontend and backend.
Co-authored-by: René Pfeuffer <rene.pfeuffer@cloudogu.com>
On windows unit tests are failing because junit checks if all @tempdir directries are empty and can be deleted after test run.
But due to opened file handles (not closed resource streams) Windows keeps files, which are "in use".
Linux is less strict in this area.
Additionally I want highlight that XMLStreamReaderImpl/XMLStreamWriterImpl from apache.xerces library (in OpenJDK11 at least) which are picked at runtime as xml parser implementation - they don't close associated resources.
BTW, I thing that relying on some runtime (sometimes - unpredictable) dependencies - is bad practice, but this it up to separate topic.
Additional fix: in IOUtil is file is locked (due to permissions or opened handle) - it will undlessly try-and-retry to delete it until end of the world, on windows.
Adds a "file lock" command that can be used to mark files as locked by a specific user. This command is implemented for git using a store to keep the locks.
Additionally, the Git LFS locking API is implemented.
To display locks, the scm-manager/scm-file-lock-plugin can be used.
Co-authored-by: Eduard Heimbuch <eduard.heimbuch@cloudogu.com>
Adds a pre receive repository hook that prevents the deletion of the default branch.
Mirrored repositories will change their default branches to another branch, when it is deleted.
Clears the external group cache whenever a user gets logged out by the logout rest method or the user gets deleted.
Co-authored-by: Eduard Heimbuch <eduard.heimbuch@cloudogu.com>
The enum fields were not searchable, because they were stored without analysation or transformation, but if an enum field was searched for within a query, the StandardAnalyzer was used. This means that the enum was stored in the index as an uppercase string, but the query searches for lowercase (the StandardAnalyzer uses a lowercase filter).
To fix this problem we are now using the KeywordAnalyzer for every non tokenized field. The StandardAnalyzer is only used for tokenized fields, which does not specify an other analyzer such code, path or id.
For enum fields we have introduced a new analyzer which uses an uppercase filter by default, this makes it possible to ignore case during search for enum fields.
The IndexOptions api has several problems:
- It is possible to open the same index with different options, which could lead to scoring problems
- If the index is already opened from another task, the options are ignored and the one from the opening task are used
- The analyzer which is derived from the options is used for every field which has not configured a specific analyzer
- This change removes the options api completely.
Co-authored-by: Konstantin Schaper <konstantin.schaper@cloudogu.com>
Id's can now be combined with more than just a repository. It is now possible to build a more complex Id such as Comment -> Pull request -> Repository. The id's now bound to a specific type. This makes it harder to accidentally use a id within an index of the wrong type.
Before this change the search uses a single index which distinguishes types (repositories, users, etc.) with a field (_type).
But it has turned out that this could lead to problems, in particular if different types have the same field and uses different analyzers for those fields. The following links show even more problems of a combined index:
https://www.elastic.co/blog/index-vs-typehttps://www.elastic.co/guide/en/elasticsearch/reference/6.0/removal-of-types.html
With this change every type becomes its own index and the SearchEngine gets an api to modify multiple indices at once to remove all documents from all indices, which are related to a specific repository, for example.
The search uses another new api to coordinate the indexing, the central work queue.
The central work queue is able to coordinate long-running or resource intensive tasks. It is able to run tasks in parallel, but can also run tasks which targets the same resources in sequence. The queue is also persistent and can restore queued tasks after restart.
Co-authored-by: Konstantin Schaper <konstantin.schaper@cloudogu.com>
Apply proxy support for jGit by extracting the required functionality from the DefaultAdvancedHttpClient into its own class HttpURLConnectionFactory. This new class is now used by the DefaultAdvancedHttpClient and jGit.
The HttpURLConnection also fixes proxy server authentication, which was non functional in DefaultAdvancedHttpClient.
The proxy support for SVNKit is implemented by using the provided method of the BasicAuthenticationManager.
For mercurial the support is configured by writing the required settings to a temporary hgrc file.
Remove non-searchable fields from syntax site, do no translate field names, use field name for title if no translation is available, refactor syntax page to respect error states
* Introduce RepositoryCoordinates
RepositoryCoordinates will be used for the enrichment of the embedded repositories of search result hits. This is required, because if we used the normal repository for the enrichment, we would get a lot of unrelated enrichers would be applied.
* Add builder method to HalEnricherContext
With the new builder method it is possible to add an object to the context with an interface as key.
* Add enricher support for embedded repository by applying enricher for RepositoryCoordinates
* Use embedded repository for avatars
The Search api is now simpler, because it provides useful defaults. Only if you want to deviate from the defaults, you can set these values. This is mostly reached by using the builder pattern. Furthermore it is now possible to configure an analyzer per field. The default analyzer is still the one which is derived from the index options, but it is possible to configure a new indexer with the analyzer attribute of the indexed annotation. The attribute allows the configuration for code, identifiers and path. The current implementation uses the same analyzer code, identifiers and path. The new implemented splits tokens on more delimiters as the default analyzer e.g.: dots, underscores etc.
Co-authored-by: René Pfeuffer <rene.pfeuffer@cloudogu.com>
Expose an api which makes it easy to detect the content type of files. The api is based on the spotter api, but does not expose spotter classes.
Co-authored-by: René Pfeuffer <rene.pfeuffer@cloudogu.com>
Add a dedicated search page with more results and different types.
Users and groups are now indexed along with repositories.
Co-authored-by: René Pfeuffer <rene.pfeuffer@cloudogu.com>
The search link of the index resource is now an array of links instead of single templated link.
The array contains one link for each searchable type.
Co-authored-by: René Pfeuffer <rene.pfeuffer@cloudogu.com>
We introduced a new annotation '@IndexedType' which gets collected by the scm-annotation-processor. All classes which are annotated are index and searchable. This opens the search api for plugins.
Add a powerful search engine based on lucene to the scm-manager api.
The api can be used to index objects, simply by annotating them and add them to an index.
The first indexed object is the repository which could queried by quick search in the header.
It was possible to download the default SCM-Manager gpg keys and overwrite them with the same raw key. This made the new key deletable. This behaviour is not longer possible.
Using a default user with a default password has the implicit risk, that this user is not changed and therefore this system can be compromised. With this change, SCM-Manager does not create the default user with the default password on startup any more, but it shows an initial form where the initial values for the administration user have to be entered by the user. To secure this form, a random token is created on startup and printed in the log.
To implement this form, the concept of an InitializationStep is introduced. This extension point can be implemented to offer different setup tasks. The creation of the administration user is the first implementation, others might be things like first plugin selections or the like.
Frontend components are selected by the name of these initialization steps, whose names will be added to the index resource
(whichever is active at the moment) and will be show accordingly.
Co-authored-by: Eduard Heimbuch <eduard.heimbuch@cloudogu.com>
